Aiming at the problems that a wide variety of Chinese medicinal materials have small samples, and it is difficult to classify the vessels of them, an improved convolutional neural network method was proposed based on multi-channel color space and attention mechanism model. Firstly, the multi-channel color space was used to merge the RGB color space with other color spaces into 6 channels as the network input, so that the network was able to learn the characteristic information such as brightness, hue and saturation to make up for the insufficient samples. Secondly, the attention mechanism model was added to the network, in which the two pooling layers were connected tightly by the channel attention model, and the multi-scale cavity convolutions were combined by the spatial attention model, so that the network focused on the key feature information in the small samples. Aiming at 8 774 vessel images of 34 samples collected from Chinese medicinal materials, the experimental results show that by using the multi-channel color space and attention mechanism model method, compared with the original ResNet network, the accuracy is increased by 1.8 percentage points and 3.1 percentage points respectively, and the combination of the two methods increases accuracy by 4.1 percentage points. It can be seen that the proposed method greatly improves the accuracy of small-sample classification.